88 research outputs found
Analysis of Different Types of Regret in Continuous Noisy Optimization
The performance measure of an algorithm is a crucial part of its analysis.
The performance can be determined by the study on the convergence rate of the
algorithm in question. It is necessary to study some (hopefully convergent)
sequence that will measure how "good" is the approximated optimum compared to
the real optimum. The concept of Regret is widely used in the bandit literature
for assessing the performance of an algorithm. The same concept is also used in
the framework of optimization algorithms, sometimes under other names or
without a specific name. And the numerical evaluation of convergence rate of
noisy algorithms often involves approximations of regrets. We discuss here two
types of approximations of Simple Regret used in practice for the evaluation of
algorithms for noisy optimization. We use specific algorithms of different
nature and the noisy sphere function to show the following results. The
approximation of Simple Regret, termed here Approximate Simple Regret, used in
some optimization testbeds, fails to estimate the Simple Regret convergence
rate. We also discuss a recent new approximation of Simple Regret, that we term
Robust Simple Regret, and show its advantages and disadvantages.Comment: Genetic and Evolutionary Computation Conference 2016, Jul 2016,
Denver, United States. 201
Fully Parallel Hyperparameter Search: Reshaped Space-Filling
Space-filling designs such as scrambled-Hammersley, Latin Hypercube Sampling
and Jittered Sampling have been proposed for fully parallel hyperparameter
search, and were shown to be more effective than random or grid search. In this
paper, we show that these designs only improve over random search by a constant
factor. In contrast, we introduce a new approach based on reshaping the search
distribution, which leads to substantial gains over random search, both
theoretically and empirically. We propose two flavors of reshaping. First, when
the distribution of the optimum is some known , we propose Recentering,
which uses as search distribution a modified version of tightened closer
to the center of the domain, in a dimension-dependent and budget-dependent
manner. Second, we show that in a wide range of experiments with unknown,
using a proposed Cauchy transformation, which simultaneously has a heavier tail
(for unbounded hyperparameters) and is closer to the boundaries (for bounded
hyperparameters), leads to improved performances. Besides artificial
experiments and simple real world tests on clustering or Salmon mappings, we
check our proposed methods on expensive artificial intelligence tasks such as
attend/infer/repeat, video next frame segmentation forecasting and progressive
generative adversarial networks
Fully Parallel Hyperparameter Search: Reshaped Space-Filling
International audienceSpace-filling designs such as Low Discrepancy Sequence (LDS), Latin Hypercube Sampling (LHS) and Jittered Sampling (JS) were proposed for fully parallel hyperparameter search, and were shown to be more effective than random and grid search. We prove that LHS and JS outperform random search only by a constant factor. Consequently, we introduce a new sampling approach based on the reshaping of the search distribution, and we show both theoretically and numerically that it leads to significant gains over random search. Two methods are proposed for the reshaping: Recentering (when the distribution of the optimum is known), and Cauchy transformation (when the distribution of the optimum is unknown). The proposed methods are first validated on artificial experiments and simple real-world tests on clustering and Salmon mappings. Then we demonstrate that they bring performance improvement in a wide range of expensive artificial intelligence tasks, namely attend/infer/repeat, video next frame segmentation forecasting and progressive generative adversarial networks
- âŠ